First-Year Effects of Early Indicator and Intervention 
Systems in Oregon 


Susan Sepanik, Pei Zhu, Miki Bairstow Shih, and Nicholas Commins August 2021 


Although Oregon has made recent gains in its overall high school graduation rate, 21 percent of public 
school students entering grade 9 in 2014 did not graduate within four years, by 2018. To improve 
graduation rates, Oregon voters approved Ballot Measure 98 in 2016 to fund dropout prevention and 
college and career readiness initiatives in high schools. Many districts used the funding to adopt an early 
indicator and intervention system (EIIS) to identify students who are not on track to graduate on time by 
monitoring related indicators, such as chronic absenteeism, disciplinary infractions, course progression, 
and academic performance, through a frequently updated data system. Districts can tailor the system by 
setting their own on-track thresholds for each indicator to identify students at risk of not graduating on 
time, assigning those students to interventions, and monitoring student response to the interventions. 


This study took advantage of the additional funding being offered to districts across the state to look 
at first-year effects on chronic absenteeism, disciplinary infractions, course progression, and academic 
performance by comparing the outcomes in 65 districts that adopted an EIIS to the outcomes in a set 
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readiness initiatives. The study offers insight into the effectiveness of early efforts to scale up EIlSs, a 
popular school-level intervention. EllIS adoption appears to have reduced the percentage of students 
who were chronically absent by 3.9 percentage points but does not appear to have had positive effects 
on the three other student outcomes during the first year: the percentage of students with disciplinary 
infractions, the percentage of grade 10 students who had acquired enough credits by the end of grade 9 
to be considered on track for on-time graduation, or the percentage of grade 11 students meeting or 
exceeding proficiency standards on state math and English language arts tests. The findings offer the 
Oregon Department of Education information on the early effects of its efforts to promote EIIS across 
Oregon. The findings can also be used by other state and district education leaders to inform their 
considerations to scale up EIIS or other similar programs. 


Although national high school graduation rates have risen over the past decade, many students still do not gradu- 
ate on time—defined as graduation within four years of entering high school—and some never earn a high school 
diploma. Fifteen percent of public school students in the United States did not graduate within four years in 2018 
(U.S. Department of Education, n.d. b). Likewise, Oregon’s graduation rate has improved in recent years, but the 
state still has one of the lowest high school graduation rates in the country. One in five Oregon public school 
students entering grade 9 in 2014 did not graduate within four years, by 2018 (U.S. Department of Education, 
n.d. b). Although some of these students graduate later or eventually earn an alternative credential, many do not. 
On average over 2013-17, 6.9 percent of young adults ages 16-24 in Oregon who 
were not attending high school had not earned a high school diploma or alterna- 


tive credential (McFarland et al., 2019). For additional information, 
including background 


Educational attainment, including on-time high school graduation, is important on the study, technical 


to future employment and earnings. In 2017 the average full-time earnings for 
young adults ages 25-34 was $26,000 for those who had not completed high 
school compared with $32,000 for high school graduates (McFarland et al., 2019). 
On-time high school graduation is also an important first step toward attaining 
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higher education credentials, which are also associated with better earnings. In 2017 young adults with an associ- 
ate degree earned an average of $39,000 a year, and those with a bachelor’s degree or higher earned an average 
of $55,000 (McFarland et al., 2019). Although many students who do not graduate on time earn alternative cre- 
dentials later, such as through the GED, they are less likely to attain postsecondary degrees and often earn lower 
wages than those with a high school diploma (Heckman et al., 2010). 


Beyond earnings, on-time high school graduation and further educational attainment influence other aspects of 
personal and social well-being. Compared with high school graduates, students who do not graduate are more likely 
to live in poverty, suffer from poor health, become incarcerated, or depend on social services (Moore, 2014). Higher 
high school graduation rates can also lead to societal benefits, such as lower crime rates, reduced public spending on 
healthcare, and greater job growth (Alliance for Excellent Education, 2013a, 2013b). Many states, including Oregon, 
have responded to the sharp contrast in long-term outcomes between graduates and nongraduates and the societal 
benefits of higher graduation rates by prioritizing efforts to boost graduation rates and on-time graduation rates. 


In November 2016 Oregon voters approved Ballot Measure 98 to provide funding to school districts for high school 
dropout prevention and college and career readiness initiatives (Oregon Department of Education, 2019). The 
measure provided $170 million in funding for the High School Success Initiative from 2017 through 2019 (Oregon 
Department of Education, 2019). During the 2017/18 school year eligible districts were directed to use High School 
Success funds to conduct needs assessments and plan interventions or program changes but were not required 
to implement these initiatives during the first year of funding. Before receiving funding in the 2018/19 school year, 
eligible districts were required to submit a High School Success Plan describing their dropout prevention plans 
and college and career readiness initiatives. The following types of programming were eligible for funding during 
2018/19: establishment and expansion of career and technical education programs; expansion of college-level 
education activities, such as Advanced Placement, dual credit, and other accelerated college credit programs; and 
dropout prevention activities designed to reduce chronic absenteeism, establish and administer data manage- 
ment systems that provide timely reports of grades, absences, and disciplinary infractions, and offer individual- 
ized supports to students in danger of not graduating (Oregon Department of Education, 2020). 


Many districts in Oregon planned to use the funding in 2018/19 to adopt an early indicator and intervention 
system (EIIS) to prevent students from dropping out of high school and to improve graduation rates (see box 1 
for definitions of key terms and box 2 for evidence of the effectiveness of EllSs). An EIIS, also referred to as an 
early warning system or early warning and intervention monitoring system, enables school staff to use data to 
identify students who do not meet on-track thresholds on indicators associated with high school graduation, such 
as attendance, behavior, and course performance. Districts can tailor their EllS by defining their own on-track 
thresholds for each indicator. After identifying students who do not meet the defined thresholds, districts can 
assign individualized interventions to these students to prevent dropout, monitor students’ response to those 
interventions, and make changes as needed. EIIS have been adopted across the country because they enable 
school districts to use real-time data to identify the students most in need of support and offer an organized 
system to manage a school’s diverse individualized services. 


Box 1. Key terms 


Baseline years. The five school years before the program year (2013/14—2017/18). 


Comparison districts. The 29 comparison districts in this study that did not use the EIIS data system managed by the Willamette 
Education Service District in 2018/19. While most districts in the state submitted High School Success Plans and received High 
School Success Initiative funding, the study team reviewed the plans of all possible comparison districts and included as compar- 
ison districts only districts that did not report plans to adopt an EIIS as part of the High School Success Initiative during that year. 
Activities listed in the comparison districts’ High School Success Plans included establishing or expanding career and technical 
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education programs, expanding Advanced Placement or dual credit programs, or implementing dropout prevention programs 
focused on boosting attendance and lowering chronic absenteeism across the school but not using data to identify and track spe- 
cific students, as is done in EIS districts. While the comparison districts did not list using any EllS components on their High School 
Success Plans, the study team was unable to examine implementation for this study and could not verify that comparison districts 
did not implement any EIIS components during the study period. 


Early indicator and intervention system (EIIS). Also known as early warning systems or early warning intervention and mon- 
itoring systems; a systematic approach to preventing dropout and supporting students’ on-time graduation. EllSs represent a 
collaborative effort by district and school staff, parents, and community organizations to use data effectively to identify students 
who are not meeting attendance, behavior, or course performance goals; assign those students to interventions that target their 
immediate and longer-term need for support; and frequently monitor their responses to those interventions and modify them 
as needed (Frazelle & Nagel, 2015). Although the basic premise was similar across districts adopting EIISs in Oregon in 2018/19, 
districts had flexibility in the design and implementation of their ElIS. This means that features of the system might have differed 
in different schools. 


EIS districts. The 65 districts across Oregon that used the EIIS data system managed by the Willamette Education Service District 
during the 2018/19 school year (the program year). The study team was unable to conduct implementation research for this study 
and could not verify how robustly EllSs were implemented in these districts. But it was ascertained that staff members at each EIIS 
district accessed the EIIS data dashboard managed by the Willamette Education Service District (n.d.) during the 2018/19 school 
year. It is possible that some EIIS districts implemented some components of an EIIS prior to the 2018/19 year. 


High School Success Initiative. An initiative that provided an additional $170 million in funding to school districts across Oregon 
from 2017 to 2019 for dropout prevention and college and career readiness initiatives (Oregon Department of Education, 2019). All 
districts received the funds for the 2017/18 school year, but to receive funds for the 2018/19 school year, districts were required 
to create a High School Success Plan that described their planned dropout prevention and college and career readiness activities. 
Since 2019, Oregon has continued and expanded the initiative. 


Outcome measures. The study examined the following outcome measures (see appendix B for more details): 

e Chronic and severe chronic absenteeism. The percentage of students who were absent for more than 10 percent of enrolled 
days (chronic absenteeism) or for more than 15 percent of enrolled days (severe chronic absenteeism). 

e Disciplinary infractions. The percentage of students who were ever suspended or expelled during the year, as reported in the 
Oregon Department of Education data. 

e Course progression. The percentage of grade 9 students who obtained at least 25 percent of the credits required for high 
school graduation, as defined by the Oregon Department of Education. Data on other measures of course performance or 
progression—such as course grades, grade point averages, or promotion to the next grade level—were not available. 

e Academic performance. The percentage of grade 11 students who scored proficient or better on the Smarter Balanced assess- 
ments in math or English language arts. This measure is not typically used as an indicator in an EIIS because data are generally 
available only once during a student’s high school years; however, stronger academic performance is a medium-term goal of 
EIISs (see figure 1 in main text). 


Program year. The 2018/19 school year. Although the High School Success Initiative provided funding from 2017 through 2019 
(Oregon Department of Education, 2019), in the 2017/18 school year districts were asked to conduct needs assessments and plan 
interventions or program changes and were not required to implement interventions. The 2018/19 school year was the first school 
year in which districts were required to submit a plan describing their dropout prevention or college and career readiness activ- 
ities and implement those activities. In that year many districts planned to use the funding to build and strengthen their EllSs. 
Because the study team was unable to conduct implementation research for this study, the actual level of implementation of EIISs 
in 2018/19 is not known. 
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Box 2. Evidence of the effectiveness of early indicator and intervention systems 


Researchers have identified key indicators of high school dropout that can be used in an early indicator and intervention system 
(EIS) to monitor students’ risk factors. Schools have little influence over some factors, such as family and student background 
characteristics and early adult responsibilities, and greater influence over others, including school engagement, behavior, and 
course performance (Hammond et al., 2007). Other research has identified three main school-related indicators that are asso- 
ciated with dropping out or not graduating high school: chronic absenteeism, disciplinary infractions, and course performance 
(Allensworth & Easton, 2005; Balfanz et al., 2014; Balfanz et al., 2007; Bowers et al., 2013; Kemple et al., 2013; Seeskin et al., 2020; 
Uekawa et al., 2010). Attendance and course performance data, in particular, appear to be strong predictors of dropout and high 
school graduation, but the evidence for disciplinary infractions is mixed (Allensworth et al., 2018). (See appendix A for more infor- 
mation on associations between these indicators and high school dropout and graduation.) 

Researchers have also tried to determine the point at which students start to fall off track from graduation. The greatest number 
of students fall off track in grade 9, followed by grade 10 (Allensworth & Easton, 2005; Seeskin et al., 2020). Based on this research, 
districts often implement a new EIIS incrementally, starting with earlier grades before expanding the program to upper grades. 

Few rigorous studies of EllSs have been conducted. The What Works Clearinghouse practice guide Preventing Dropout in Second- 
ary Schools recommends early warning systems as an important tool for dropout prevention but notes that the evidence to support 
them is still slight (Rumberger et al., 2017). One study found that ElISs can reduce chronic absenteeism and course failure in the first 
year of adoption but did not find that they affect grade point average, suspensions, or credits earned (Faria et al., 2017). Another 
study of a school reform model that included an ElIS component did not find impacts of the model on the attendance, behavior, or 
course performance indicators individually but did find impacts on a composite measure of the three indicators (Corrin et al., 2016). 

The key components of a typical EllS are shown in figure 1 in the main text, which explains how an EIIS could affect short-, 
medium-, and long-term student outcomes. When all EllIS components are implemented at a school, it is hypothesized that overall 
rates of chronic absenteeism and disciplinary infractions will likely decrease and course performance will likely improve. If students 
are more likely to attend school (measured as fewer chronically absent students), engage in classes (measured as fewer disciplinary 
infractions), and progress in their coursework (measured as fewer course failures), they are expected to strengthen their academ- 
ic skills and build proficiency in core subjects such as math and English language arts (measured as performance on standardized 
assessments). The short-term effects might include better engagement in school and increased persistence, meaning students stay in 
school longer and do not drop out. It is also expected that that improvement in short-term outcomes such as course performance will 
increase progression in school (students progressing to the next grade level and not being held back due to course failures). Stronger 
academic performance and increased persistence and progression could ultimately lead to improved high school graduation rates. 


The districts in this study that adopted EIISs used a data system that was created by a consortium of Oregon 
school districts and education service districts. The system is managed by the Willamette Education Service 
District (n.d.) and enables educators to monitor indicators of students’ attendance, disciplinary infractions, and 
course performance and to regularly identify students who are not meeting the district-set thresholds for being 
on track for high school graduation. District or school staff members who are adopting an EIIS also create a menu 
of available interventions that are mapped to the selected indicators. For example, afterschool math tutoring 
could benefit students who are flagged for receiving a D in a math course, whereas a first period check-in could 
support a chronically absent student. A response team made up of teachers, administrators, and counselors from 
the school and district meets at designated intervals throughout the school year. At each of these meetings, the 
response team uses the data system dashboard to identify students who fall below one or more indicator thresh- 
olds. The response team then uses the intervention menu to identify the best available supports for students, 
depending on their specific needs (Frazelle & Nagel, 2015). 


Although the Oregon Department of Education (ODE) promoted districts’ use of an EIIS via High School Success Initia- 
tive funding, districts decided on the design and implementation of their EllS. Figure 1 describes ideal EllS implemen- 
tation. Some Oregon districts might have diverged from the model or implemented only some of the components.' 


1. The study team was unable to collect implementation data, so the study cannot report on the level of implementation for each district. 
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Figure 1. Key components and outcomes of early indicator and intervention systems to improve high school 
graduation rates 


Components Short-term Medium-term Long-term 
* School districts use the Willamette Education Service outcomes outcomes outcome 
District’s data dashboard to identify students who ¢ Reduced © Stronger overall ¢ Improved 
do not meet district-defined thresholds for chronic academic high school 
attendance, behavior, or course performance absenteeism performance graduation rate 
Each high school maps a menu of available ¢ Reduced © Increased rate of ¢ Improved 
interventions for each indicator, which can include: disciplinary persistence and career 
e Attendance: daily check-ins and policy to call infractions grade progression readiness 
home after each absence ¢ Improved 
¢ Behavior: peer mediation groups » course B » 
© Course performance: afterschool tutoring or performance 
homework help 
e Multiple indicators: individual coaching 
A response team of school-specific teachers, 
administrators, and counselors meets periodically 


during the school year and assign interventions 
to students according to their specific needs 
The response team monitors each student’s 
progress and adjusts interventions as needed 


Source: Authors’ adaptation of Faria et al. (2017). 


ODE, with assistance from the Regional Educational Laboratory (REL) Northwest, sought to understand whether 
ElISs, in their first year of adoption, supported local improvements in attendance, behavior, course progression, 
and academic performance of high school students in Oregon. These outcome measures are similar to the indica- 
tors used in an EIIS to identify students for support and allowed the study team to examine early effects of EIISs 
on key indicators of high school dropout.? 


Research questions 


The study systematically examined the first-year effects of ElIS adoption in Oregon districts on student outcomes 
during 2018/19. Using ODE data, the study compared student outcomes in 65 districts that adopted at least some 
features of an EllS under the High School Success Initiative with student outcomes in 29 similar comparison dis- 
tricts that used the funding for other purposes, including college and career readiness initiatives or other dropout 
prevention activities. However, the study team was not able to obtain comprehensive information about the 
actual implementation of ElISs and other initiatives during the program year (see box 3 for a summary of the 
study data sources, sample, methods, and limitations). 


The study examined the following research questions: 


1. What were the early effects of EllSs on chronic absenteeism and disciplinary infractions across all high school 
students (grades 9-12 combined) and in each of grades 9-12? 


2. What were the early effects of ElISs on course progression in grade 9 and academic performance in grade 11? 


3. What were the early effects of ElISs on subgroups of students who would likely have been identified by an 
EIIS as at risk of not graduating on time in the year prior to the study: students who were severely chronically 
absent, who had disciplinary infractions, or who had not accumulated enough credits by the end of grade 9 to 
be considered on track to graduate (grade 10 students only)? 


2. The exception is that an EllS tends to measure course performance, often determined by whether a student has any Ds or Fs in courses. 
The study team was unable to collect course grades from ODE and instead looked at measures of grade 9 course progression and aca- 
demic performance on grade 11 state standardized assessments. See box 1 for more information about the outcome measures. 
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Research question 1 explores the effects of EllSs on chronic absenteeism and disciplinary infractions across all 
high school students in all grades combined and by grade level. An EllIS response team at a school generally meets 
regularly throughout the school year and identifies students who have recently failed to meet a threshold (a 
student who is absent for more than 10 percent of days during the first semester or a student who has a failing 
grade in math in the third quarter). The EIIS is meant to identify and support any student who starts to fall off 
track regardless of when that happens in high school and whether the student has been off track in the past. The 
way districts rolled out their EllS could have affected the outcomes. For example, the study looked only at the 
first year of EIIS adoption. Districts can roll out the program incrementally, focusing on earlier grades in the early 
years of adoption. Moreover, the typical rollout of an ElIIS emphasizes the need to identify students for interven- 
tions as early as possible to ensure that students receive support before they get too far behind. It is therefore of 
interest to see whether different patterns of findings emerge across grade levels. 


Research question 2 explores the effects of ElISs on course performance using two proxy measures: a course 
progression indicator that captures students’ overall credit accumulation, available only for grade 9 students, and 
students’ performance on standardized math and English language arts tests, available only for grade 11 students. 
While these two measures were the best available proxies for students’ course performance, neither measure 
directly captures students’ performance in specific courses, one of the hypothesized short-term outcomes of an 
EIIS (see figure 1). In addition, the measures are considered lagging indicators, since it is generally hard to affect 
course performance in the early stage of EIIS implementation. 


Because the goal of an EIIS is to identify students who are not meeting attendance, behavior, or course perfor- 
mance goals and to provide interventions to help those students meet these goals, an EIIS might be expected to 
be especially beneficial to the students it identifies as not meeting these goals. However, because the study team 
did not have access to EIIS data, the team could not directly explore this question. Instead, to address research 
question 3, the study team looked at the effects of an EIIS on subgroups of students who might be expected to be 
identified by the EIIS: those who were severely chronically absent, had disciplinary infractions, or did not accumu- 
late the required grade 9 credits in the year prior to the study, when students were in grade 10. 


The study findings can be used to inform ODE’s decisions on whether districts should be encouraged to priori- 
tize an EIIS over other options that can be funded by the High School Success Initiative. Other state and district 
stakeholders and community partners can also use the findings to inform their decisions about implementing 
statewide policies to promote EIISs or similar programs. 


Box 3. Data sources, sample, methods, and limitations 


Data sources. The study used primarily student- and school-level administrative data provided by the Oregon Department of 
Education. Additional school-level data were collected from the U.S. Department of Education’s Common Core of Data database 
(U.S. Department of Education, n.d. a). Data from the early indicator and intervention system (EIIS) managed by the Willamette 
Education Service District (n.d.) were used to identify the EIIS program districts. 


Sample. The sample consisted of 65 EIIS districts and 29 non-EllS districts in four regions in Oregon.’ On average in EIIS districts, 
about 31 percent of the student population were racial/ethnic minority students, about 3 percent were English learner students, 
and 14 percent had Individualized Education Programs (see table B2 in appendix B). The average number of high schools per 
district was 1.4, and average enrollment in EIIS district high schools was 688 students. Approximately 78 percent of EIIS districts 
were in rural areas, and 11 percent were eligible for Title | funding (financial assistance to districts and schools with high numbers 
of children from low-income families). Relative to other school districts in Oregon, the EIIS districts served a higher proportion of 
English learner students. The high schools in EIIS districts were generally larger and had higher student-teacher ratios than the 
average district in Oregon. Despite these differences in district structural characteristics, on average the EIIS districts were similar 
to other districts in Oregon in student absenteeism, behavior, course progression toward graduation, and academic achievement, 
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as measured in 2017/18, the year before EIISs were rolled out. Although the EIIS districts all used the same EIIS data system, how 
each district implemented an EIIS, relative to the key components shown in figure 1 in the main text, is unknown. 

The study team also selected 29 comparison districts from the 75 eligible districts in the same four Oregon regions that 
matched best with the EIIS districts. Districts were matched on the baseline level and trend of all outcome measures, as well as 
district student composition in the last baseline year, such as average school enrollment and percentages of racial/ethnic minority 
students. Districts were also matched on structural characteristics such as high school graduation rates and school locale (see 
appendix B for a full list of matching variables). EIS districts and comparison districts were found to be very similar on student 
composition and performance on student outcomes in the last baseline year (see table B3 in appendix B). 


Methodology. The study used a comparative interrupted time series (CITS) design to evaluate the effect of EllSs on districts. CITS 
evaluates program effects by examining whether school districts that implemented an intervention (in this case, an EllS) deviated 
from their baseline trends by a greater amount than a group of similar comparison districts. This methodology was chosen for 
three primary reasons: 
¢ Oregon’s High School Success Initiative caused a substantial influx of funding and other support for EllSs from the Oregon 
Department of Education during the program year. 
e Alarge number of districts adopted an EIIS with funding from the initiative. Other similar districts chose instead to adopt other 
initiatives, including other dropout prevention activities or college and career readiness initiatives. 
¢ School records for multiple school years were available for all districts in Oregon, allowing the study team to select and match 
districts based on their characteristics and trends across multiple school years prior to EIIS adoption. 
The study team conducted multiple sensitivity tests to explore whether the comparison districts provided the best match for 
the EIIS districts and for the robustness of the study findings (see appendix B). One sensitivity test result deviated from the study 
findings and is discussed in the Findings section. 


Limitations. The study had two key limitations. First, because of the project timeline, the study could examine and report on the 
effect of EllSs only on short-term and some medium-term outcomes after one year of EIIS adoption. It could take more than one 
year for these outcomes to be fully realized. The effects might differ after a few years of implementation, once school and district 
staff members have more experience with the system. Another important limitation is the lack of implementation information. 
Although the study design did not include a full implementation study, it did include plans to collect descriptive information to 
provide context for the findings on student outcomes. These plans included collecting information on the frequency of data pulls 
by each EIIS district from the EIIS data system managed by the Willamette Education Service District (n.d.) to understand how 
educators interacted with the data system. Interviews with local education service district staff members were also planned, to 
help assess the level of implementation at EIIS districts and whether any similar activities took place at comparison districts. The 
COVID-19 pandemic, ensuing school closures, and shifts to online learning made it impossible to collect these data. Without this 
information the study team could not assess whether the study findings represent a test of fully implemented EllSs or whether 
comparison districts were using similar systems to monitor and target students in need. 


Note 


1. There are five geographic regions in Oregon. The Central region was excluded from the analysis because no district in this region implemented an EIIS 
during the 2018/19 school year. 


Findings 

The findings in this section relate to the first-year effects of ElIS adoption during the 2018/19 school year. 

During the 2018/19 school year adoption of early indicator and intervention systems appears 

to have reduced the overall percentage of high school students who were chronically absent by 

3.9 percentage points 

In the years before EllSs were widely adopted, the percentage of chronically absent students rose in both EIIS and com- 
parison districts (figure 2). However, during the program year districts that adopted an EIIS appear to have reduced the 


percentage of chronically absent students more than comparison districts did, as shown by the difference between 
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Figure 2. During the 2018/19 school year districts adopting early indicator and intervention systems (EIIS) 
appear to have reduced the overall percentage of high school students who were chronically absent, 
2013/14-2018/19 

= EIIS district trend/actual == Comparison district trend/actual 


eee EIIS district trend projected °* Comparison district trend projected 
@ Actual outcome @ Projected outcome based on trend 


Percent of students chronically absent (all grades combined) Comparison district difference 
between actual and projected outcome: 
35 0.3 percentage point 


EIIS district difference between 
actual and projected outcome: 
—3.5 percentage points** 


30 
25 
20 
2013/14 2014/15 2015/16 2016/17 2017/18 2018/19 
Before EIIS adoption First year of 
EIS adoption 


** Significant at p < .01 


Note: The difference between the actual outcomes (@) and the projected outcomes (@) is calculated separately for the EIIS districts and the comparison 
districts. Each data point reflects the outcomes from the end of the respective school year. The vertical dashed line indicates that the program started 
after the 2017/18 school year and occurred during the entire 2018/19 school year. Data are pooled across grades 9-12, and all values are aggregated to 
the district level. See table C1 in appendix C for the details of these results. 


Source: Authors’ analysis based on student-level data from the Oregon Department of Education and school-level data from U.S. Department of Educa- 
tion (n.d. a). 


projections of the baseline trend into the program year (dashed blue line for EIIS districts and dashed orange line for 
comparison districts) and actual outcomes (the solid lines). In the year after EllSs were adopted, the actual percent- 
age of chronically absent students in EIIS districts (the blue dot in 2018/19) was 3.5 percentage points lower than the 
projected baseline trend for that year (the blue diamond). In contrast, in comparison districts the actual percentage of 
chronically absent students was 0.3 percentage point higher than the projected baseline trend. In other words, in the 
program year the actual percentage of chronically absent students was lower than would have been predicted based 
on the baseline trend in EIIS districts while the actual percentage was about the same as predicted in comparison 
districts. The estimated effect of ElIS adoption on chronic absenteeism was —3.9 percent (the 0.3 percentage point 
difference for the comparison districts subtracted from the —3.5 percentage point difference for the EIIS districts).? 


Adoption of early indicator and intervention systems appears to have reduced chronic absenteeism 
rates by more than 5 percentage points for students in grade 10 and by more than 6 percentage 
points for students in grade 11 


The effect of EIIS adoption on chronic absenteeism appears to be concentrated in grades 10 and 11. EllS adoption 
appears to have reduced chronic absenteeism rates in grade 10 by 5.1 percentage points, to 27.5 percent, and in 
grade 11 by 6.4 percentage points, to 29.9 percent (table 1). Although the percentage of chronically absent stu- 
dents also appears to have declined in grades 9 and 12, the differences are not statistically different from zero and 
should be interpreted with caution. 


3. This estimated effect is statistically significant at the 5 percent level, meaning that if ElIS had had no effect on chronic absenteeism, the 
probability of observing the estimated effect of -3.9 percentage points or larger would be no more than 1 percent. 
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Table 1. During the 2018/19 school year adoption of early indicator and intervention systems (EIISs) appears to 
have reduced the percentage of high school students who were chronically absent overall in all grades and in 
grades 10 and 11 


Actual Estimated mean Estimated 
Grade level mean? without EIISs° effect® 
All grades 29.7 33.6 —-3.9* 
Grade 9 23.7 24.0 -0.3 
Grade 10 27.5 32.3 —5.1* 
Grade 11 29.9 36.3 —6.4** 
Grade 12 37.9 42.3 4.4 


* Significant at p < .05; ** significant at p < .01. 


Note: All values are aggregated to the district level. A two-tailed t-test was applied to each estimated effect of EllSs. Rounding might cause slight dis- 
crepancies in calculating the sums and differences. See table C1 in appendix C for the details of these results. 


a. Realized mean outcome levels for districts that adopted an EIIS in the 2018/19 school year. 
b. Differences between the actual mean and the estimated effect. 
c. Estimated effect of EIIS adoption in the 2018/19 school year. 


Source: Authors’ analysis based on student-level data from the Oregon Department of Education and school-level data from U.S. Department of Educa- 
tion (n.d. a), for school years 2013/14—2018/19. 


Adoption of early indicator and intervention systems appears to have reduced severe chronic absenteeism rates by 
an average of 3.3 percentage points across all grade levels (grades 9-12). EIS adoption appears to have reduced 
the percentage of severely chronically absent students. In the EIIS districts about 17 percent of high school stu- 
dents were severely chronically absent during the 2018/19 school year (table 2). EIIS adoption appears to have 
reduced overall severe chronic absenteeism by 3.3 percentage points. 


This reduction does not appear to apply equally to all grades. EllS adoption appears to have reduced the percent- 
age of severely chronically absent students by a statistically significant 5.2 percentage points in grade 10 and a 
statistically significant 5.6 percentage points in grade 11. In contrast, the apparent reductions for grades 9 and 12 
are not statistically different from zero. 


Table 2. During the 2018/19 school year, adoption of early indicator and intervention systems (EIISs) appears 
to have reduced the percentage of high school students who were severely chronically absent overall across 
grades and in grades 10 and 11 


Actual Estimated mean Estimated 
Grade level mean? without EIISs° effect® 
All grades 16.7 20.0 —3.3** 
Grade 9 12.4 12.8 -0.4 
Grade 10 15.0 20.1 —5:2** 
Grade 11 17.2 22.9 —5.6*** 
Grade 12 22.6 24.7 21 


** Significant at p < .01; *** significant at p < .001. 


Note: All values are aggregated to the district level. A two-tailed t-test was applied to each estimated effect of EllSs. Rounding might cause slight dis- 
crepancies in calculating the sums and differences. See table C1 in appendix C for the details of these results. 


a. Realized mean outcome levels for districts that adopted an EIIS in the 2018/19 school year. 
b. Differences between the actual mean and the estimated effect. 
c. Estimated effect of EIIS adoption in the 2018/19 school year. 


Source: Authors’ analysis based on student-level data from the Oregon Department of Education and school-level data from U.S. Department of Educa- 
tion (n.d. a), for school years 2013/14—2018/19. 


REL 2021-097 9 


During the 2018/19 school year adoption of early indicator and intervention systems does not appear 
to have affected the overall rate of disciplinary infractions across all grades and seems to have 
increased the rate of disciplinary infractions for students in grade 9 


About 6 percent of high school students across EIIS districts were suspended or expelled at least once during 
the 2018/19 school year. ElIS adoption does not seem to have affected the overall rate of disciplinary infractions 
across all grades or the rate for grades 10, 11, and 12 separately (table 3).* 


In grade 9, districtwide ElIS adoption appears to have led to a 2.8 percentage point increase in the proportion of 
students suspended or expelled, resulting in a grade 9 infraction rate of 7.7 percent in the average EIIS district. 


The grade 9 effect results largely from the comparison districts’ lower-than-expected levels of disciplinary infrac- 
tions (see table C1 in appendix C for detail), which implies that either student behavior changed in these compari- 
son districts of its own accord or that comparison districts’ approaches were more effective than EIIS adoption at 
reducing grade 9 suspensions and expulsions. The measure itself—percentage of suspensions and expulsions per 
year—cannot distinguish changes in student behavior from changes in disciplinary policy and practices. Compar- 
ison districts might have been less likely than EIIS districts to suspend or expel grade 9 students because student 
behavior improved in those districts. Alternatively, disciplinary practices at comparison district schools could 
have changed in the program year, leading to fewer suspensions and expulsions without any changes in student 
behavior. The study team had planned to conduct retrospective interviews with district education service leaders 
to understand what approaches or activities took place at the comparison districts during the program year, but 
COVID-19 restrictions and subsequent school closures prevented the collection of such information within the 
study time frame. 


Table 3. During the 2018/19 school year adoption of early indicator and intervention systems (EIISs) does not 
appear to have affected the percentage of students with disciplinary infractions across all grades but does 
appear to have increased the percentage of grade 9 students with disciplinary infractions 


Actual Estimated mean Estimated 
Grade level mean? without EIISs° effect® 
All grades 5:5. 5.2 0.3 
Grade 9 77 4.9 2.8* 
Grade 10 6.7 6.7 0.0 
Grade 11 4.6 5.3 —0.6 
Grade 12 3.0 3.5 —0.6 


* Significant at p< .05. 


Note: All values are aggregated to the district level. A two-tailed t-test was applied to each estimated effect of EllSs. Rounding might cause slight dis- 
crepancies in calculating the sums and differences. See table C1 in appendix C for the details of these results. 


a. Realized mean outcome levels for districts that adopted an EIIS in the 2018/19 school year. 
b. Differences between the actual mean and the estimated effect. 
c. Estimated effect of EIIS adoption in the 2018/19 school year. 


Source: Authors’ analysis based on student-level data from the Oregon Department of Education and school-level data from U.S. Department of Educa- 
tion (n.d. a), for school years 2013/14—2018/19. 


4. Note, however, that one of the sensitivity checks produced an estimated effect of 1.8 percentage points, indicating that EIIS adoption 
might have increased the rate of infractions by 1.8 percentage points (see table B7 in appendix B). This sensitivity analysis used an 
alternative method to select comparison districts and included fewer EIIS districts. 
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During the 2018/19 school year adoption of early indicator and intervention systems does not appear to 
have affected course progression for grade 9 students or academic performance for grade 11 students 


In Oregon grade 9 students are considered on track to graduate if they earn at least 25 percent of the total credits 
needed to graduate high school by the end of grade 9. In the 2018/19 school year 82 percent of grade 9 students 
in the average EIIS district met this course progression goal compared with an estimated 79 percent of students 
in the absence of EIIS (table 4). Thus, ElIS adoption does not appear to have affected the overall percentage of 
grade 9 students who were considered on track to graduate in the first program year.° 


Additionally, EllS adoption in Oregon does not appear to have affected academic performance in grade 11, defined as 
the percentage of students who scored at or above the proficiency level on the Smarter Balanced math assessment. 
On average, the proportion of students who scored at or above the proficiency level in math was about 31 percent 
across EIIS districts in the 2018/19 school year. Although EIIS districts did show a sizable difference between the 
actual (31 percent) and projected (26 percent) percentage of students proficient on the Smarter Balanced math 
assessment in 2018/19, a similar difference was observed for the comparison districts (effect size of 4.9; see table 
C1 in appendix C). This suggests that there were statewide shifts in math performance levels coinciding with the 
program year and therefore that EllIS adoption did not appear to have improved average student math achievement. 


Similarly, ElIS adoption does not appear to have significantly affected the percentage of grade 11 students meeting 
proficiency standards in English language arts. On average, the proportion of students who scored at or above 
the proficiency level in English language arts was 69 percent across sample EIIS districts in the first year of EIIS 
adoption (see table 4). The difference between the actual and projected percentage of students who scored at 
or above the proficiency level on the Smarter Balanced English language arts assessment was similar for both EIIS 
districts and comparison districts (effect size of —0.2; see table C1 in appendix C). This finding suggests that EIIS 
adoption did not improve average student English language arts proficiency. 


Table 4. During the 2018/19 school year adoption of early indicator and intervention systems (EIISs) does not 
appear to have affected the percentage of students who met course-progression or academic performance 
goals 


Actual Estimated mean Estimated 
Outcome mean? without EIISs° effect® 
Percentage of grade 9 students meeting course-progression goals4 81.7 79.3 2.4 
Percentage of grade 11 students proficient or above on the Smarter 
Balanced math assessment 31.2 26.4 4.9 
Percentage of grade 11 students proficient or above on the Smarter 
Balanced English language arts assessment 68.9 69.1 -0.2 


Note: All values are aggregated to the district level. The Smarter Balanced assessment was first administered in 2014/15, and data on proficiency include 
2015/16—2018/19. A two-tailed t-test was applied to each estimated effect of EllSs. Rounding might cause slight discrepancies in calculating the sums and 
differences. See table C1 in appendix C for the details of these results. 


a. Realized mean outcome levels for districts that adopted an EIIS in the 2018/19 school year. 

b. Differences between the actual mean and the estimated effect. 

c. Estimated effect of EIIS adoption in the 2018/19 school year. 

d. Students met course-progression goals if they had obtained 25 percent of the credits needed for graduation by the end of grade 9. 


Source: Authors’ analysis based on student-level data from the Oregon Department of Education and school-level data from U.S. Department of Educa- 
tion (n.d. a), for school years 2013/14—2018/19. 


5. This estimated effect is not statistically significant at the 5 percent level, meaning that there is more than a 5 percent probability that 
the actual effect is zero. 
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It is not surprising that EllSs did not affect students’ academic performance during the first year of ElIS adoption, 
because longer program implementation periods are generally required for a program to affect student academic 
outcomes. (This is also why these two measures are listed as medium-term outcomes in figure 1.) Nonetheless, 
the study team explored this hypothesis because early assessment of EIIS program effects on student perfor- 
mance could provide useful confirmation that it might take longer for a program to impact student academic 
outcomes. 


There was no evidence that adoption of early indicator and intervention systems affected any student 
outcomes for students who did not meet attendance, behavior, or course-progression goals in the 
previous year 


The study team analyzed the effects of EllSs on subgroups of students who would likely have been identified 
through an EIIS during the year before adoption to see whether EIIS adoption affected the outcomes of stu- 
dents who were struggling with attendance, behavior, or course performance before the 2018/19 program year. 
These subgroups included students who were severely chronically absent, had disciplinary infractions, or failed 
to accumulate enough credits by the end of grade 9 to be considered on track to graduation. Overall, the study 
team found no evidence indicating that districtwide EIIS adoption benefited any of these student subgroups (see 
appendix B for information about these subgroups and appendix C for findings on these subgroups). 


Implications 


This study provides an initial look at the first-year effects of EIIS adoption on policy-relevant outcomes in 65 
Oregon school districts. Four main implications can be drawn from the findings for ODE, education service dis- 
tricts, and participating districts to consider as they adopt, implement, or scale an EIIS. 


The mixed findings on the first-year effects of early indicator and intervention systems suggest that 
the Oregon Department of Education and participating districts might want to identify the indicators 
and interventions that could be most beneficial to students 


Districts and schools have flexibility in choosing which indicators to use in their ElIS. Thus, it would be possible 
for districts and schools to include some indicators on which an EIIS might have positive effects, such as chronic 
absenteeism, while excluding other indicators on which an EIIS might have negative effects, such as disciplinary 
infractions. More research would be needed to determine the effectiveness of this type of approach and to iden- 
tify which indicators would be most effective in specific contexts. 


During the first year of districtwide adoption in Oregon, EllISs appear to have reduced the proportion of students 
who were chronically absent beyond any reduction that occurred in comparison districts, making it worth explor- 
ing whether using an EIIS to target and support students with attendance issues might alleviate the problem of 
chronic absenteeism in Oregon high schools and elsewhere. In Oregon approximately 30 percent of high school 
students are chronically absent and about 17 percent are severely chronically absent (see table B2 in appendix B), 
while across the country more than 20 percent of high school students are identified as chronically absent (U.S. 
Department of Education, 2019). Evidence demonstrates that missing school interferes with students’ academic 
success and graduation (Balfanz & Byrnes, 2012). As a result, chronic absenteeism is a major area of concern 
across the country (U.S. Department of Justice et al., 2015). 


The study did not find positive effects in EIIS districts on other outcome measures and found one negative effect: 
students in EIIS districts were more likely than students in comparison districts to have at least one disciplinary 
infraction in grade 9. As noted earlier, the evidence for disciplinary infractions as a predictor of high school gradu- 
ation is more mixed than the evidence for attendance and course performance. Some districts, including Chicago 
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Public Schools, have chosen not to use disciplinary infractions in their EllS, whereas other districts do use disci- 
plinary infractions (Allensworth et al., 2018). ODE might want to explore this further, gathering more evidence on 
the effectiveness of ElISs on different student outcomes across Oregon to provide guidance to school districts on 
whether to consider dropping disciplinary infractions from their EllIS or changing the interventions they use when 
students are identified as meeting or exceeding the disciplinary infraction threshold. Further, school districts in 
Oregon that adopt or plan to adopt an EIIS can gather their own data on student indicators and outcomes to 
support decisions on which indicators and which interventions to include. 


Additional effort might be needed to ensure that the early indicator and intervention systems 
initiative supports grade 9 students as they transition into high school 


Leaders at ODE and in participating districts might want to pay particular attention to the effects of EIIS adoption 
in grade 9 and on grade 9 student outcomes. Because the study was unable to include research on EIIS imple- 
mentation, there is no information on whether Oregon districts implemented EIISs differently in grade 9 than in 
other grades. But the study did find that the grade 9 outcome for chronic absenteeism was less promising than 
the grade 10 and 11 outcomes. Although EIIS adoption appears to have positive effects on chronic absenteeism 
on average across all grades and particularly for grade 10 and 11 students, the same was not true for grade 9 
students. 


EIIS adoption did not appear to affect disciplinary infractions across all grades or in grade 10, 11, or 12 separately 
(although one of the sensitivity tests for the robustness of the finding did reveal an increase in disciplinary infrac- 
tions across grades). However, ElIS adoption appears to have increased the rate of disciplinary infractions for 
grade 9 students compared with districts that used the High School Success Initiative funding in other ways. (Com- 
parison districts showed lower than expected rates of disciplinary infractions, suggesting that their approaches 
were more effective than ElISs at reducing disciplinary infractions.) It might be worth exploring whether EIISs 
have the potential to reduce grade 9 disciplinary infractions when implemented fully or whether other interven- 
tions might be more effective in supporting grade 9 students with behavior issues. Because disciplinary infrac- 
tions include both student behavior and staff member decisions and actions in identifying and responding to 
infractions, a finding of more disciplinary infractions could mean either that students are having more behavior 
violations or that staff are reacting to student behavior more punitively. For instance, it is possible that some staff 
members might be more likely to record disciplinary infractions in districts that are implementing EIISs in the 
expectation that such actions could result in more EIIS behavioral supports for such students. 


EIIS adoption also does not seem to have affected the percentage of students who were considered on track 
to graduate at the end of grade 9. Accumulating the requisite number of credits to be on track to graduate by 
the end of grade 9 is one of the strongest indicators of future graduation (Allensworth & Easton, 2005; Kemple 
et al., 2013). As students transition into high school, they often experience less support, larger class sizes, and less 
personal attention from teachers than they had in middle school, in addition to encountering more demanding 
course work, higher academic expectations, and stronger social pressures. As a result, students are more likely 
to disengage from high school in grade 9 than later in their high school careers (Allensworth & Easton, 2007). 
Disengagement during grade 9 is highly correlated with dropping out or failing to graduate (Roderick et al., 2014). 
Students who fall behind in grade 9 have a harder time recovering credits and face a greater risk of dropping out 
(Allensworth & Easton, 2007). Therefore, to support students’ later educational attainment, it might be partic- 
ularly important for districts to use EllSs to support students who struggle with key risk factors during the first 
year of high school. For instance, districts that implement EIISs could include additional training for staff who 
work with grade 9 students to ensure that data from the middle grades (grades 6—8) are accessed and properly 
reviewed for all students entering grade 9. 
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To better understand what led to this study’s findings and identify potential ways to strengthen early 
indicator and intervention systems (ElISs) programming, the Oregon Department of Education and 
participating districts might want to study EIIS implementation 


Because of the impacts of the COVID-19 pandemic, the study team was unable to collect implementation and 
contextual information that could help assess the extent of EIIS implementation in Oregon. To fully understand 
and accurately interpret this study’s findings—and to strengthen EIIS adoption—ODE and participating districts 
might want to assess program implementation and identify challenges that could affect success. They might also 
want to consider a study that reviews the alternative types of activities that have been funded by the High School 
Success Initiative and that have been implemented by the comparison districts in this study. In 2019 Oregon leg- 
islators passed the Student Success Act, which extended funding for the High School Success Initiative for several 
years (Oregon Department of Education, 2019). The continuity of funding might encourage districts to continue 
or begin implementing EIISs, so it could be helpful to develop a better understanding of how the degree of imple- 
mentation and its context might affect student outcomes. 


To understand the effects of a mature early indicator and intervention system (EIIS) program, the 
Oregon Department of Education and participating districts might want to consider a follow-up study 
of the longer-term effects of EllSs on student outcomes 


This study took advantage of Oregon’s introduction of the High School Success Initiative to look at early effects of 
ElISs adopted using the initiative funding, but this study is only a first step in assessing the effectiveness of these 
systems. Although this study did not find that ElISs had positive effects on outcomes other than chronic absentee- 
ism (that is, on disciplinary infractions, course progression, or academic performance), 2018/19 was the first year 
of EIIS adoption for many districts. It might take longer to see effects on the other outcomes, especially academic 
performance. EIISs are large-scale interventions with complex components, including a data system, a menu of 
interventions, a fully articulated process to identify students and assign them to interventions, and a system to 
monitor and modify interventions. Districts might need several years to fully and effectively implement an EIIS. 
In addition, EllSs are meant to support students throughout high school, so it might take more than one year 
before supports lead to observable positive changes in behavioral and academic outcomes for students. Thus far, 
research on EllSs has looked only at first-year outcomes (Faria et al., 2017). Long-term follow-up research could 
assess the effects of a fully implemented EIIS. Ideally, a follow-up study would look at a mature system imple- 
mented for at least four years and follow participating students from grade 9 through high school graduation to 
measure whether EllSs met their long-term outcome goal of improving high school graduation rates. 
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